Summary

• Adds a StripThinkingFilter (lmms_eval/filters/transformation.py) that strips <think>...</think> / <thinking>...</thinking> reasoning blocks from model generations, delegating to the existing lmms_eval.api.reasoning.strip_reasoning_tags helper. Handles both full tag pairs and the common chat-template case where only the closing tag appears in the completion (opening tag was prefilled into the prompt).
• Registers it in FILTER_REGISTRY as "strip_thinking".
• Adds TaskConfig.auto_strip_thinking: bool = False — strictly opt-in per task (or via task overrides), so scoring inputs for existing tasks/users are bit-identical by default.
• In evaluator.evaluate(), when a task enables it, StripThinkingFilter is inserted at the front of each of the task's existing filter pipelines — chained, so the answer-extraction filters (take_first, regex, multi-choice) actually receive the stripped text, and no extra scored filter key is created. Auto-strip is skipped when the task/CLI already configures reasoning_tags, which strips at scoring time — running both would double-strip.

Why

Reasoning models increasingly emit <think> blocks before their final answer. Without stripping, answer-extraction filters see the whole reasoning chain and mis-score otherwise-correct answers. This gives tasks a one-line opt-in for the standard fix.

Open question for maintainers

This overlaps conceptually with the existing reasoning_tags mechanism (which rewrites filter keys at scoring time). I kept this as a filter because it composes with per-task pipelines, and gated the two against each other — but happy to consolidate onto one mechanism if you prefer.

Test plan

• python3 -m py_compile on all changed files
• Functional test: filter strips full tag pairs, handles the prefill/closing-tag-only case, passes plain text through unchanged
• TaskConfig().auto_strip_thinking defaults to False
• End-to-end pipeline test (test/eval/test_strip_thinking_filter.py): with the filter enabled, a decoy answer inside <think> is ignored and the real answer is extracted by the downstream regex filter; filtered_resps contains no extra strip_thinking key; 99 filter/eval tests pass
• ruff check introduces zero new findings vs. pre-change baseline

Review pass (2026-07-06)

Self-review found the original wiring added the strip step as a sibling filter ensemble — ensembles don't chain, so extraction filters never saw stripped text and the new ensemble's list-shaped output would itself get scored (crashing e.g. MMMU's process_results). Follow-up commit rewires it to prepend into each existing pipeline (as described above), flips the default to opt-in, adds the reasoning_tags gating, and documents two known limitations in strip_reasoning_tags (case-sensitive tags; unclosed opening tag passes through).